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DETAILED ACTION 

Response to Amendment 

1 . The amendments, filed on 05/1 1/2009, have been entered and made of record. Claims 9, 20, 
and 31 have been cancelled and claims 1-8, 10-19, 21-30 and 32-36 have been amended. 
Claims 1-8, 10-19, 21-30 and 32-36 are pending. 

Response to Arguments 

2. Applicant's arguments with respect to claims 1-8, 10-19,21 -30 and 32-36 have been 
considered but are moot in view of the new ground(s) of rejection. 

Drawings 

3. The drawings were received on 05/1 1/2009. This drawing which includes Figs 3a, 3b, 4, 5, 
6a and 6b are replacing the original sheets including Figs 3a, 3b, 4, 5, 6a and 6b. These 
drawings are accepted and made of record. 

Specification 

4. The substitute specification received on 05/1 1/2009 has been accepted, entered and made of 
record. Therefore, the objection has been withdrawn. 
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5. The text of those sections of Title 35, U.S. Code not included in this action can be found in a 
prior Office action. 

6. Claims 1-3, 5-8, 10, 14-19, 21, 25-30, 32, and 36 are rejected under 35 U.S.C. § 103(a) as 
being unpatentable over Lee (US 6,778,207) in view of Chang (US 7,307,654). 

Regarding claim 1, Lee discloses a method for generating a wide image video sequence 
(provide piecewise coverage of a panorama with overlapping visual fields) (1 :63-66) using a 
device having at least two video cameras substantially co-located in a predetermined 
relationship to each other such that there will be an overlap (i.e. overlap region 300, Fig 4) 
between images (i.e. regions P and Q, Fig 4) from the respective cameras (i.e. three cameras 
91, 92, and 93 arranged so that their respective fields of view overlap) (5:8-21 and Fig 2), 
said method comprising the step of : 

- forming a synthetic image from images of the respective cameras by: (i) identifying 
corresponding parts in overlapping parts of the images (as illustrates in Fig 4, two 
overlapping regions P and Q overlap in a region 300 as the regions P and Q have 
already been warped to a common surface so that features in each image being 
identified and coincided) (5:48-6:6) and (ii) determining the relation between the 
respective coordinates for the pixels in the individual cameras and in the synthetic 
image (the pixel properties of overlapping image regions are blended 220, as the 
blending of properties (e.g., R, G, B, and/or intensity) may be achieved by weighted- 
averaging the properties of overlapping regions in the merged images, wherein any or 
all of the pixel properties may be weight-averaged within the domain and as shown in 
Figs 7-9, the application of the transform to Image 12 to yield image region 12', and 
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the four registration points of the object images 317 and 319 coincide at 321 in the 
overlapping region 320) (5:40-6:59). 
- synchronously recording video sequences using each of said at least two video 
cameras (three cameras 91, 92, and 93, Fig 2); and 

generating a wide image video sequence by combining said synchronously recorded 
video sequences (an array of fixed digital cameras is mounted on an arrangement to 
provide piecewise coverage of a panorama with overlapping visual fields) (1 :63-66). 
Lee does not disclose calculating calibration parameters from said relation, said 
calculated calibration parameters being unique for the at least two cameras and their current 
location as related to the object being recorded. However, Chang discloses (iii) calculating 
calibration parameters (compute both internal and external calibration parameters) from said 
relation, said calculated calibration parameters being unique for the at least two cameras and 
their current location as related to the object being recorded (compute both internal and 
external calibration parameters of first camera 14 and second camera 16, as the internal 
calibration parameters include but are not limited to the focal length and lens distortion of 
each camera and the external calibration parameters include the relative position and 
orientation of first and second cameras 14, 16 with respect to one another) (Figl and 3-5, and 
5:12-53, Chang); 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the control module 12 as taught by Chang into Lee's digital 
camera, as the suggestion/motivation would have been to enable the controller which is 
capable of performing the calibration, analysis, and interpolation computations to derive a 
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synthesized image, to create synthesized images allows the user to select the most 
satisfactory angle to view the scene in order to better ensure that the specific region of 
interest at a particular time instance is accessible to the user (3:29-46 and 10:50-64, Chang). 

Regarding claim 2, Lee discloses the synchronously recorded video sequences are stored 
in a memory means (a processor 1 10, which having a memory, Fig 1) (5:8-20 and claim 
number 8, Lee). 

Regarding claim 3, the limitations of claim 1 are taught above, Chang discloses the 
synchronously recorded video sequences are used concurrently for generating the wide image 
video sequence (at step 130, first camera 14 captures a first subject image of the portion of 
subject 52 and planar background 18 positioned within first projection profile 40 and in step 
132, which occurs simultaneously with step 130 , second camera 16 captures a second subject 
image or video of the portion of subject 52 and planar background 18 positioned within 
second projection profile 42) (5:54-6:7, Chang). 

Regarding claim 5, Lee discloses the wide image video sequence (provide piecewise 
coverage of a panorama with overlapping visual fields) (1:63-66) is stored on a memory 
means (a processor 110, which having a memory, Fig 1) (5:8-20 and claim number 8, Lee). 

Regarding claim 6, the limitations of claim 1 are taught above, Lee does not disclose a 
detailed calibration process according to this claim. However, Chang discloses calculating 
the calibration (calibration process 102, Fig 3); comprises the following steps: 

a. starting of calibration process (step 110, Fig 3) (at 1 10, a predesigned calibration 
pattern 50 is displayed in front of planar background 18, i.e. on front planar 
surface 44) (4:56-5:11, Chang); 
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b. synchronizing the sequences from each of the at least two cameras (blocks 112 
and 116 occur simultaneously, and blocks 1 14 and 118 occur simultaneously or 
near simultaneously) (4:56-5:1 1, Chang); 

c. computing inter-image projective transformations (step 122, Fig 3) (the 
correspondence mapping and geometric parameters of planar background 18 
determined at 120 are utilized to compute both internal and external calibration 
parameters of first camera 14 and second camera 16) (5:32-53, Chang); 

d. using the transformations to refer each image to a common reference frame (step 
120, Fig 3) (at 120, CPU 30 compares the robust features from the first and 
second images to the known characteristics of calibration pattern 50 and performs 
a correspondence mapping and the correspondence mapping entails locating each 
captured robust characteristic of calibration pattern 50 in first image 14 and noting 
the spatial relationship of the captured robust characteristics) (5:12-31, Chang); 

e. selecting a real or virtual reference camera such that certain lines on the pitch or 
stadium arc substantially horizontal and substantially parallel in the wide image 
(step 122, Fig 3) (the overall spatial relationship of first camera 14, second 
camera 16, and planar background 18 is used to determine a first coordinate 
system with respect to first camera 14 and a second coordinate system with 
respect to second camera 16) (5:32-53, Chang); 

f. selecting a rectangular region of interest within the wide image (the 
correspondence mapping and geometric parameters of planar background 18, Fig 
1, as the planar background 18 is determined at 124 based upon the calibration 
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parameters of each camera) (5:32-53, Chang); and 
g. storing the computed values resulting from the calibration process to be used as 
the calibration parameters (once the two coordinate systems are derived, the 
calibration process is complete and the first subject image or video is recorded by 
first camera 14 and transferred to CPU 30 via first video capture device 32) (5:32- 
6:7, Chang). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Lee's device, 
as the suggestion/motivation would have been to enable the controller which is capable of 
performing the calibration, analysis, and interpolation computations to derive a synthesized 
image, to create synthesized images allows the user to select the most satisfactory angle to 
view the scene in order to better ensure that the specific region of interest at a particular time 
instance is accessible to the user (3:29-46 and 10:50-64, Chang). 

Regarding claim 7, the limitations of claims 1 and 6 are taught above, Lee does not 
disclose the step of determining lens distortion parameters for each camera and correcting 
radial distortion in each image. However, Chang discloses the step of determining lens 
distortion parameters for each camera, and correcting radial distortion in each image (adapted 
to stretch the synthesized video image on the video display 20 to remove any parallax 
distortions) (3:47-67, Chang). 

Regarding claim 8, the limitations of claims 1 and 6 are taught above, Chang discloses 
the step of selecting non-linear distortion parameters to reduce perspective distortion of the 
image (the internal calibration parameters include but are not limited to the focal length and 
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lens distortion of each camera) (5:32-53, Chang). 

Regarding claim 10, the limitations of claim 1 are taught above, Chang discloses the step 
b (step 1 14 and 118, Fig 3) is performed automatically by an algorithm for identification of 
corresponding features in concurrent video images and the coordinates for the corresponding 
features are input via a computer means (the second image is transferred from second camera 
16 to CPU 30 via second video capture device 34. CPU analyzes the second image for the 
robust features of calibrated pattern 50) (4:56-5:1 1 and Fig 3, Chang). 

Regarding claim 14, this claim differs from claim 1 only in that the claim 1 is a method 
claim whereas claim 14 recites similar features in an apparatus format. Thus the apparatus 
claim 9 is analyzed and rejected as previously discussed with respected to claim 1 above. 

Regarding claims 15 and 16, these claims recite same limitations as claims 2 and 3, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 2 and 3 above. 

Regarding claims 17, 18, 19, and 21, these claims recite same limitations as claims 6, 7, 
8, and 10, respectively. Thus they are analyzed and rejected as previously discussed with 
respect to claims 6, 7, 8, and 10 above. 

Regarding claim 25, this claim recites same limitations as claim 1. Thus it is analyzed 
and rejected as previously discussed with respect to claim 1 above. 

Regarding claims 26 and 27, these claims recite same limitations as claims 2 and 3, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 2 and 3 above. 
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Regarding claims 28, 29, 30, and 32, these claims recite same limitations as claims 6, 7, 
8, and 10, respectively. Thus they are analyzed and rejected as previously discussed with 
respect to claims 6, 7, 8, and 10 above. 

Regarding claim 36, this claim recites substantially the same limitations as claim 1. Thus 
it is analyzed and rejected as previously discussed with respect to claim 1 above. 

7. Claim 4 is rejected under 35 U.S.C. § 103(a) as being unpatentable over Lee in view of 
Chang, and further in view of Alonso (US 6,445,293). 

Regarding claim 4, the limitations of claim 1 are taught above, Lee and Chang do not 
disclose the wide image video sequence is transmitted live. However, Alonso discloses the 
wide image video sequence is transmitted live (the camera system sets the front camera as 
output image and transmits live video out from this camera) (3:56-65, Alonso). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the imaging pickup device as taught by Alonso into Lee and 
Chang's device, as the suggestion/motivation would have been to guarantee that the camera 
system will remain in the alarm state in case of an emergency (1 :36-39 and 3:41-45, Alonso). 

8. Claims 11,13, 22, 24, 33, and 35 are rejected under 35 U.S.C. § 103(a) as being unpatentable 
over Lee in view of Chang, and further in view of Matsumoto (US 2003/0071906). 

Regarding claim 1 1 , the limitations of claim 1 are taught above, Chang discloses the 
method according to claim 1 which comprises the following steps: 

a. apply the calculated calibration parameters (at 1 10, a predesigned calibration 
pattern 50 is displayed in front of planar background 18, i.e. on front planar 
surface 44) (4:56-5 : 1 1 and step 1 10, Fig 3, Chang); 
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c. retrieving one new image from each camera (at 112, first camera 14 captures a 
first image of calibration pattern 50 and at 1 16, second camera 16 captures a 
second image of calibration pattern 50) (4:56-5:1 1 and steps 1 12 and 116, Fig 3, 
Chang); 

d. selectively updating the parameters needed to transform intensities in one or more 
of the cameras to eliminate visible seams (correspondence mapping entails 
locating each captured robust characteristic of calibration pattern) (5:12-31 and 
step 120, Fig 3, Chang); 

e. selectively adjusting intensities in the images from one or more of the cameras (a 
similar correspondence mapping procedure is completed using the second image 
to determine the geometric parameters of planar background 1 8 with respect to 
second camera 16) (5:12-31 and step 120, Fig 3, Chang); 

f. creating the current seamless, wide image from the current images from each 
camera (map corresponding robust features to determine geometric parameters of 
planar background, since the calibration pattern 50 is a planar surface located 
upon planar background 1 8, the geometric parameters of planar background with 
respect to first camera 14 are directly computed from the mapped correspondence 
information) (5:12-31 and step 120, Fig 3, Chang); and 

g. outputting the image to a display or to a memory storage area (storage device 37, 
Fig 1) (after all calibration parameters are determined, the overall spatial 
relationship of first camera 14, second camera 16, and planar background 18 is 
determined at 124 based upon the calibration parameters of each camera and the 
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image is to a display or to a memory means) (5:32-53, Chang). 

Lee and Chang do not disclose a calibration process is repeated until the end of the 
sequence is reached or return to step b until end of generation of the video sequence. 
However, Matsumoto disclose a calibration process is repeated until the end of the sequence 
is reached (the calibration process is repeated at a predetermined interval until the release 
switch (SW1) 126 or the release switch (SW2) 127 is pressed, Figs 7-9) ([0183], Matsumoto) 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Lee's device, 
as the suggestion/motivation would have been to enable the controller which is capable of 
performing the calibration, analysis, and interpolation computations to derive a synthesized 
image, to create synthesized images allows the user to select the most satisfactory angle to 
view the scene in order to better ensure that the specific region of interest at a particular time 
instance is accessible to the user (3:29-46 and 10:50-64, Chang). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Matsumoto into Lee and 
Chang's device, as the suggestion/motivation would have been to enable that the more 
accurate correction data can be obtained ([0183], Matsumoto). 

Regarding claim 13, the limitations of claims 1 and 1 1 are taught above, Chang discloses 
the new images from each video camera are read from a memory means (storage device 37, 
Fig 1) (after all calibration parameters are determined, the overall spatial relationship of first 
camera 14, second camera 16, and planar background 18 is determined at 124 based upon the 
calibration parameters of each camera and the image is to a display or to a memory means) 
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(5:32-53, Chang). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Lee's device, 
so as the image data (e.g., video images, image shape and color information) stored in 
storage device 37 will be available for use at a later time (6:63-7:20 Chang). 

Regarding claims 22 and 24, these claims recite same limitations as claims 1 1 and 13, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 11 and 13 above. 

Regarding claims 33 and 35, these claims recite same limitations as claims 11 and 13, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 11 and 13 above. 

9. Claims 12, 23 and 34 are rejected under 35 U.S.C. § 103(a) as being unpatentable over Lee in 
view of Chang and Matsumoto, and further in view of Alonso (US 6,445,293). 

Regarding claim 12, the limitations of claims 1 and 1 1 are taught above, Lee and Chang 
do not disclose the new images from each camera are read from live sources, each such 
source comprising a video camera. However, Alonso discloses the new images from each 
camera are read from live sources, each such source comprising a video camera (the camera 
system sets the front camera as output image and transmits live video out from this camera) 
(3:56-65, Alonso). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the imaging pickup device as taught by Alonso into Aagaard 
and Chang's device, so as the camera system will be capable to used by the main system to 
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increase system versatility and can get any advanced feature shown in the main video system 
(3:41-45, Alonso). 

Regarding claims 23 and 34, these claims recite same limitations as claim 12. Thus they 
are analyzed and rejected as previously discussed with respect to claim 12 above. 

Conclusion 

10. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure: 

- Ivanov et al. (US 6,9 1 1 ,995) provide a system and method for segmenting a video of 
a scene so that various depths can be detected; 

- Peterson (US 7,386, 188) provides a methods, systems, and apparatus, including 
computer program products, for merging images of segments of a view to form a 
panoramic image; and 

- Peleg et al. (US 6,665,003) provide a new and improved system and method of 
generating and displaying stereoscopic panoramic images. 

1 1 . Applicant's amendment necessitated the new ground(s) of rejection presented in this Office 
action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until 
after the end of the THREE-MONTH shortened statutory period, then the shortened statutory 
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period will expire on the date the advisory action is mailed, and any extension fee pursuant to 
37 CFR 1 . 1 36(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of 
this final action. 

12. Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Kent Wang whose telephone number is 571-270-1703. The examiner 
can normally be reached on 8:00 A.M. - 5:30 PM (every other Friday off). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Sinh Tran can be reached on 57 1 -272-7564. The fax phone number for the 
organization where this application or proceeding is assigned is 571-270-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status information 
for unpublished applications is available through Private PAIR only. For more information 
about the PAIR system, see http://portal.uspto.gov/external/portal/pair. Should you have 
questions on access to the Private PAIR system, contact the Electronic Business Center 
(EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer 
Service Representative or access to the automated information system, call 800-786-9199 (IN 
USA OR CANADA) or 571-272-1000. 
/Tuan V Ho/ 

Primary Examiner, Art Unit 2622 
KW 

2 July 2009 



